Data collection is exploding with the confluence of Internet connectivity, ubiquitous computing devices, increasing processing power, decreasing data storage cost and space, etc. The term “big data” has been coined to define large collections of data that are complex and difficult to process and analyze using conventional systems and methods. It is known in the art that data collection systems feed “control systems,” “expert” systems, and accounting systems to make sure that a certain precise activity/flow is followed, over and over again. In (most of) these data collection systems the intent is to remove “fallible, error prone, and devious” human intervention from the system to ensure the system operates “as designed,” even though it may be operating below optimal and thus subpar performance. The link between system performance and those responsible for it has been severed, with the only connection being that those responsible may receive alerts, alarms, or other event warnings to say the system is deviating from the designed flow. To find out if the system is suboptimum or needs improvement, a special “off-line” analysis or study needs to be performed. To improve this process from the original design is intended to be very difficult.
Most off-line analytics (i.e. big data) that look at the system to improve it take large amounts of raw data and look for trends, and if the trend points to improvement by statistical methods, the system is “upgraded” or a new algorithm is added. But, by and large the system still operates intentionally disconnected from humans and new target comparisons. In the old world of slow technology change, lacking links to social networks, and reduced complexity systems for laypeople, maybe that worked, but business and people now demand real-time performance systems that allow them to see all data related to target and thus make changes to get to the best in class.